Repository navigation
fix(snowflake): look up last modified timestamps in each table's database - #6132
nicksspirit merged 3 commits into
Conversation
…base SnowflakeEngineAdapter.get_table_last_modified_ts queried the unqualified INFORMATION_SCHEMA.TABLES view, which only contains objects from the connection's current database. Tables in any other database were never found, so the freshness signal raised MissingSourceError for models that read external tables from another database. Group the tables by database and query each database's own INFORMATION_SCHEMA.TABLES. The query is now built with sqlglot instead of string formatting, so table and schema names are escaped as literals. Signed-off-by: Dhruv Gupta <todhruvg@gmail.com>
|
Hey @DG47 — thanks for fixing this. Querying each database's own INFORMATION_SCHEMA is the right fix, and the new test covers the RAW vs ANALYTICS case from the issue. One edge case: the tables are grouped by Could you key the groups on the catalog identifier itself, so quoted and unquoted names stay separate? A small test that passes both Thanks again for taking the time to put this up. |
…ied lookup Group the tables by the catalog identifier instead of its name. In Snowflake an unquoted raw is the database RAW while a quoted "raw" is a different database, so grouping by name sent both to whichever information schema came first. Signed-off-by: Dhruv Gupta <todhruvg@gmail.com>
|
Thanks @nicksspirit, good catch. Updated in c7a1528: the groups are now keyed on the catalog identifier (quoting included) instead of its name, so |
|
One test is failing on windows but it's more so a dependency update thing not a huge blocker |
2 participants
Description
Fixes #6131.
SnowflakeEngineAdapter.get_table_last_modified_tsreadLAST_ALTEREDfrom the unqualifiedINFORMATION_SCHEMA.TABLESview. In Snowflake that view only lists tables in the session's current database, so tables in any other database were never returned. Thefreshness()signal then raisedMissingSourceErrorfor models that read external tables from a different database than the gateway's (for exampleRAWvsANALYTICS).This change groups the requested tables by database and queries each database's own
INFORMATION_SCHEMA.TABLES("RAW".INFORMATION_SCHEMA.TABLES, and so on). TheTABLE_CATALOGpredicate is no longer needed because each view is already scoped to one database. The BigQuery implementation groups per dataset in the same way. The query is now built with sqlglot rather than string formatting, so table and schema names are escaped as string literals. Tables without a catalog still go to the current database's information schema.Test Plan
test_get_table_last_modified_ts_multiple_catalogstotests/core/engine_adapter/test_snowflake.py. It requests three tables across two databases and checks that one query per database is issued and the timestamps are combined. It fails onmain(one unqualified query) and passes with this change.pytest tests/core/engine_adapter -m "not docker and not remote" -n 8: 714 passed, 3 skipped.pytest -n 8 -m "fast and not web and not github and not dbt and not jupyter and not cicdonly and not isolated": 2436 passed, 6 failed. The 6 failures are the pyodbc connection-string tests, which fail locally onmaintoo because unixODBC is not installed.SKIP=prettier,eslint pre-commit run --all-files: ruff, ruff-format, mypy and valid migrations all pass.The Snowflake freshness integration tests are skipped for Snowflake (
_skip_snowflakeintest_freshness.py), and I did not run this against a live Snowflake account.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO